MCEETYA 

Performance Measurement and Reporting Taskforce 

National Sample Assessment Protocols 

PURPOSE 

These protocols represent a working guide for planning and implementing national 
sample assessments in connection with the national Key Performance Measures 
(KPMs). 

The protocols are intended for agencies involved in planning or conducting national 
sample assessments and personnel responsible for administering associated tenders 
or contracts, including the: 

• Performance Measurement and Reporting Taskforce (PMRT) and its sub- 
groups 

• Benchmarking and Educational Measurement Unit (BEMU) 

• PMRT Secretariat 

• assessment contractors 

• assessment liaison personnel for schooling sectors in the states and 
territories. 

BACKGROUND 

The protocols in this document refer to national sample assessments conducted in a 
three-year cycle in: 

• science literacy (samples of Year 6 students) 

• civics and citizenship (samples of Year 6 and Year 1 0 students) 

• information and communication technology (ICT) literacy (samples of Year 6 
and Year 10 students) 

These assessments form part of a program to monitor progress towards the National 
Goals for Schooling in the Twenty-First Century, endorsed in April 1999 by the 
Ministerial Council on Education, Employment, Training and Youth Affairs 
(MCEETYA). 

PMRT reports to MCEETYA on a range of issues including the development and 
maintenance of KPMs as the basis for national reporting in a set of agreed areas. 

Ministers have agreed to report on progress towards the achievement of the National 
Goals in the following priority areas: 

• literacy 

• numeracy 

• science 

• ICT 

• civics and citizenship education. 




Ministers have also noted the need to develop indicators of performance in enterprise 
education. 

Currently, annual assessments of literacy and numeracy occur for the full cohort of 
students in Years 3, 5 and 7. In addition, Australia participates in international 
assessments in reading, mathematics literacy and scientific literacy for 15-year-old 
students as part of the Organisation for Economic Co-operation and Development 
(OECD) Programme for International Student Assessment (PISA). The protocols 
presented in this document do not refer to the annual literacy and numeracy 
assessments or to the PISA assessments. 

The protocols for the national sample assessments are presented under four major 
headings: 

• Management, Liaison and Communication 

• Development and Conduct of Assessments 

• Reporting 

• Data Storage and Access. 

MANAGEMENT, LIAISON AND COMMUNICATION 
PROTOCOLS 

Management 

PMRT has responsibility, on behalf of MCEETYA, for national sample assessments. 
The KPM Sub-group acts as the steering committee while day-to-day management of 
an assessment is carried out by BEMU. 

Assessment contractors are engaged to carry out tasks associated with the 
development, conduct and reporting of a given national sample assessment. BEMU 
administers contracts through a designated project officer. 

A national sample assessment review committee is established for each assessment 
to provide the assessment contractors with feedback on draft assessment materials 
from curriculum, psychometric and equity perspectives. 

The Director of BEMU chairs each national sample assessment review committee 
with executive support from the PMRT secretariat. Membership consists of: 

• Curriculum experts in the assessment area nominated by 

■ each state and territory government 

■ the Australian government 

■ the Independent Schools Council of Australia (ISCA) 

■ the National Catholic Education Commission (NCEC) 

• An educational measurement specialist (who may be one of the above 
representatives) 

• Specialists in Indigenous education and education of students with a 
language background other than English (LBOTE) — an additional person in 
each category if the expertise is not available through the curriculum 
specialists 

• BEMU and PMRT representatives. 

The review committee examines draft assessment instruments to ensure they are of 
appropriate difficulty, and are valid, free of bias and accessible to all participating 
students including: 

• students with an Indigenous background 



• students with a language background other than English 

• students with disabilities 

• students in metropolitan, provincial and remote areas 

• boys and girls. 

The review committee also examines assessment instruments for stereotyping or 
possible offence based on people’s sex, language, culture, ethnicity, religion, 
disability, geographic location or socioeconomic background. 

Following review by the assessment area review committee, drafts of assessment 
frameworks, assessment items and tasks, assessment forms and associated 
materials are submitted to the KPM Sub-group for approval. 

Liaison and communication 

The PMRT Chair makes formal communication with the Chief Executive Officers of 
government departments, Catholic education offices and associations of independent 
schools in the states and territories to announce the dates of the assessment, 
request the participation of schools that may be selected in the sample and call for 
the appointment of liaison officers. Normally, this communication occurs towards the 
end of the year prior to the year of testing. 

Systems are responsible for ensuring that schools participate. 

For each assessment, BEMU prepares an information statement, including sample 
items, for parents, school principals and school staff. The information statement is 
published on the MCEETYA website and made available to the Chief Executive 
Officers of government departments, Catholic education offices and associations of 
independent schools in the states and territories for distribution to schools. 

Liaison officers are appointed in each state and territory to assist the assessment 
contractors in: 

• identifying schools that should not participate in the sample because of 
participation in the previous year or other extenuating circumstances 

• securing the participation of schools in the testing 

• communicating with schools or school systems as necessary in the conduct 
of the assessment. 

Each of the government, Catholic and independent sectors in each state and territory 
may nominate a liaison officer. The Australian government also nominates a liaison 
officer to enable it to assist as required and to be kept informed of progress. 

BEMU is responsible for informing the liaison officers of their role prior to any contact 
with the sampled schools. 

Liaison officers receive copies of all materials sent to schools. 

Assessment contractors provide telephone and email access for schools and school 
systems to clarify procedures associated with administration of assessments 
(including trial processes). 

Upon endorsement of the assessment report by PMRT, the Chair notifies the Chief 
Executive Officers of government departments, Catholic education offices and 
associations of independent schools in the states and territories of the availability of 
the report. 



Quality assurance 

Assessment contractors in collaboration with BEMU establish and document quality 
assurance processes for all phases of an assessment as outlined in the contract. 
Information about the quality of the assessment is included in the contractor’s 
technical report. 

DEVELOPMENT AND CONDUCT OF ASSESSMENTS 

Timing 

National sample assessments are conducted in on a rolling triennial basis as follows: 

• Science literacy (Year 6) from 2003 

• Civics and citizenship (Year 6 and Year 1 0 students) from 2004 

• ICT literacy (Year 6 and Year 10 students) from 2005. 

National sample assessments are typically administered in the latter half of October; 
however, the testing window may extend from September to November for some 
assessments. 

The timing is such as to avoid conflict with: 

• school vacations across Australia 

• state and territory assessment programs 

• international assessment programs. 

The dates of the assessment are announced in the year prior to the assessment 
year. 

Schools selected in the sample are notified before the end of August of the 
assessment year. 

Development of instruments 

Development of instruments (assessment forms) starts from a conceptual framework 
for the assessment area in the form of a set of domains. The conceptual framework 
is founded on a mapping of existing curriculum and assessment across states and 
territories. 

A progress map is developed showing a developmental sequence for each domain. 
The levels are conceived as representing what can be expected from students at key 
stages of schooling. 

The assessment framework derives from the progress map, showing the distribution 
of item types for each level in each domain. The levels included in the assessment 
framework are centred on those expected for the year level of the sample. 

In developing assessment frameworks, BEMU and assessment contractors arrange 
and document consultation processes with stakeholders. 

Assessment items and tasks are developed in accordance with the assessment 
framework. 

Sufficient assessment items and tasks are included to enable: 

• linking between Year 6 and Year 10 results 

• linking between successive cycles of assessment 

• an assessment kit (see below) for release to schools after the assessment. 



Assessment forms are part of a package that includes administration, marking, 
analysis and reporting processes and materials. 

Drafts of assessment frameworks, items and tasks, trial results, test forms and 
associated materials are reviewed by the assessment area review committee. The 
final assessment domain and assessment materials are endorsed by the KPM Sub- 
group. 

Trial of assessment items and tasks 

Processes to trial assessment items and tasks are negotiated by the contractor with 
each of the states or territories taking part. Such negotiation includes consideration 
of: 

• the proposed duration of the assessment, including number of tasks 

• the role of teachers and any requirements for professional development and 
training 

• administrative demands on schools 

• timing of the trialling 

• compatibility with current assessment policies and practices within the states 
and territories 

• resource implications for schools 

• data collection from students or teachers (e.g. surveys or interviews) 

• consultation/communication arrangements with other key stakeholders in the 
project. 

Trials are structured to allow calibration of items on a single scale. 

Trial samples are designed to be sufficiently representative of the target population to 
yield adequate information on students’ responses across the expected range of 
levels, language development, geolocation, interests, socio-cultural background and 
other relevant variables. 

Sample design 

The sample for the assessment includes adequate representation of the states and 
territories as well as government, Catholic and independent schools, although 
comparisons are not made between the different sectors. 

The sample size is sufficient to allow comparisons between subgroups according to 
each of the following variables separately (no simultaneous comparisons): 

• sex (male, female) 

• Indigenous status (Indigenous, non-lndigenous) 

• geographic location (metropolitan, provincial, remote and very remote) 

• parental education (levels to be decided) 

• parental occupation (levels to be decided) 

• language background (LBOTE) 

Over-sampling may be necessary for some subgroups. 

The sample should not include schools that were included in the previous year. 

Guidelines for selecting the sample, including sampling processes for small states 
and territories, are included in the contract specification. 



Background data 

Data on student background characteristics (other than geographic location) are 
collected in accordance with the nationally consistent definitions as set out in the 
MCEETYAdata implementation manual. 

For Year 10 students, background data are collected on the assessment forms. For 
Year 6 students, the data are collected by the school or school system and 
transmitted to the assessment contractor. [Refer to MCEETYA data implementation 
manual.] 

Geographic location 

For Year 6 students, geographic location is determined according to the location of 
the school and is derived from the MCEETYA Schools Geographic Location 
Database, which is prepared by the Australian Government Department of 
Education, Science and Training (DEST) and updated annually. A copy of the latest 
database is provided to the contractor for use with Year 6 students. 

For Year 10 students, geographic location is determined according to the home 
location of the student. BEMU provides the contractor with appropriate wording for 
the student questionnaire to collect the necessary information. BEMU also provides 
coding procedures to enable coding of the address information to the four levels set 
out above. 

Test administration 

The assessment contractor supplies each school in the sample with a comprehensive 
assessment package containing all of the information and materials necessary to 
administer the assessment, including: 

• assessment forms 

• background information for staff, students and parents 

• guidelines for grouping students for assessment administration 

• equipment or special conditions required 

• instructions for administration and return of completed forms 

• detailed and explicit instructions for those administering the assessment to 
students 

• training material for staff administering the assessment 

• instructions for recording absence, withdrawal or exemption 

• special procedures for students with learning or physical disability or 
language difficulties 

• practice materials. 

Within the guidelines provided, administration of the assessment forms to students is 
a matter for school organisation. 

The assessment contractor provides funds to support small schools in arranging 
teacher release or support staff as necessary to administer the assessment. 

The assessment contractor monitors the actual testing process by means of visits to 
a sample of the schools during the testing period. 

Assessment contractors include advice in the school administration guidelines on 
how principals can meet the requirements of the relevant privacy legislation so that 
students, parents and schools are properly informed of who will have access to the 
data and how the information from the testing will be reported. 



Assessment contractors document the reasons given by schools for declining to 
participate in the assessment. 

Participation in assessment: absence, exemption and withdrawal 

Assessment processes and instruments are designed to encourage maximum 
participation by all students. 

Every student in classes that are selected in the sample takes part in the 
assessment, except where the student is: 

• withdrawn from the assessment at the request of parents/caregivers or 

• granted exemption because of disability, unfamiliarity with the English 
language or special circumstances. 

Where feasible, special considerations may be given to students with disabilities and 
students with a language background other than English. These special 
considerations are variations of the standard assessment conditions to provide 
students with the opportunity to display what they know and can do without being 
limited or impeded by their disability or language background. Such special 
considerations will be implemented only where they do not change what is being 
assessed or introduce assessment bias. Such considerations would not be expected 
to go beyond what is routinely available to the student in the classroom setting. 

Marking 

Marking schemes are prepared by the contractor during the development of the 
assessment items. 

Refinement of the marking scheme may sometimes be necessary during the trial 
process or the early stages of marking the national sample assessments but the 
same marking scheme is applied to all students' assessment responses. 

BEMU arranges for participation in the marking by representatives from each state 
and territory as well as ISCA and NCEC. 

Standards 

Standards are the result of national collaborative processes and agreed key 
decisions. 

National sample assessment standards differ from the national literacy and numeracy 
benchmarks in that: 

• The benchmarks describe nationally agreed minimum acceptable standards 
for aspects of literacy and numeracy at particular year levels. That is, they 
represent the minimum acceptable standard without which a student will have 
difficulty making sufficient progress at school. 

• A national sample assessment standard defines the performance of a student 
at the particular year level who is considered to be proficient in the curriculum 
area being assessed. 

The process of identifying standards is iterative and uses stakeholders’ views in an 
empirical-judgmental technique. Typically this would need to occur only during the 
first round of national sample assessments in each area. 

BEMU assumes responsibility for the standard-setting process and: 

• selects a suitable panel including experienced teachers 

• leads the panel through the process 

• documents the process and its outcomes. 



The proposed standards are endorsed by PMRT. 

Assessment kits 

The contractor develops assessment kits for use by schools. 

Assessment kits allow schools to administer, mark the publicly released items and 
gauge their own students’ performances in relation to the national standards. 

An assessment kit includes: 

• assessment forms parallel to those used in the sample assessment, 
comprised of the public release items 

• background information for staff, students and parents 

• details of equipment or special conditions required 

• instructions for administration 

• marking guide 

• interpretation guide for results 

• practice materials. 

REPORTING 

Nature of reports 

The primary audiences for the reports from the national sample assessments are 
educators and the public. The PMRT may release the main findings in various forums 
and documents tailored to suit specific purposes and audiences. 

Reports on sample assessments are expected to be published within six months of 
return of the completed assessment forms. 

Reports are of three types: 

• national reports for wide distribution 

• technical reports for MCEETYA and PMRT 

• school reports. 

National reports 

National reports for the public domain are concise and readable by a wide audience. 

Data are presented in national reports in such a way that they can be clearly 
understood by the lay reader and the message easily grasped without the need for 
complex statistical understanding. 

No information is reported publicly that permits the identification of individual students 
or schools without their permission. The publication of data for subgroups at a 
state/territory level occurs only where the confidentiality of individuals, small groups 
or institutions is protected. 

National reports should include the following: 

• the purpose of the assessment 

• the relevant national goals for the assessment 

• any legislative requirements which may impact on the data 

• a description of the assessment forms and procedures 

• a description of the review processes for items and standards, the trial 
processes, and of the sampling process and outcomes 



• discussion of limitations of the data 

• all results with estimates of statistical uncertainty 

• a description of quality assurance processes including editing, compliance 
checks and audits 

• justification of the validity and reliability of the results from the national 
assessment. 

Technical reports 

The primary audience for technical reports is MCEETYA, through PMRT, and its 
committees and sub-groups. 

Technical reports are to be comprehensive and need to document the procedures 
and decisions applied during the calibrations of the relevant scales and in the 
preparation of the national report. The technical report will be critical to ensuring that 
future rounds of assessments can replicate the analyses and statistics. Technical 
reports will include but not be limited to: 

• details of the assessment forms and procedures 

• details of the review processes for items and standards 

• details of trial processes 

• details of the sampling process and outcome 

• discussion of limitations of the data 

• all results with estimates of statistical uncertainty 

• details of quality assurance processes including editing, compliance checks 
and audits 

• justification of the validity and reliability of the results from the national 
assessment. 

School reports 

School reports include summary information on the performance of individual 
students and the school in relation to the national results. 

Approval and publication 

Drafts of reports are approved by MCEETYA before publication. 

The BEMU works with the contractor in reviewing and revising draft reports. Issues 
that cannot be resolved are referred to the KPM and DCR Subgroups of PMRT. 

The national report is published in hard copy, in limited numbers, and on the 
MCEETYA website. 

DATA STORAGE AND ACCESS 

The full data set is available to PMRT once entry is complete and verified. 

All data and information about the procedures used for compilation of the data are 
stored systematically and securely by the assessment contractor. 

The assessment contractor makes technical information about sampling or statistical 
analysis accessible for data auditing and verification purposes. 

Ownership of data generated in national sample assessments and all associated 
rights remain with MCEETYA, unless other arrangements have been agreed within 
an assessment contract. 



Access to assessment data is determined by MCEETYA. No information identifying 
individuals or schools is available to any person or agency other than the person or 
school concerned. 

Access to data for research purposes is decided by PMRT upon application. 
Permission to access data is conditional upon PMRT being satisfied that: 

• the risk of damage to individuals or institutions is minimal 

• researchers have access to the skills needed to carry out and interpret the 
results of the planned analyses 

• hypotheses that define the analysis are clearly stated in advance 

• copies of all reports will be supplied to PMRT 

• full acknowledgment of MCEETYA will be made in any form of related 
publication. 



